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The Intelligence Community (IC) is 
well known as an espionage ser- 
vice. Much less well known is the 
fact that it is one of the world's big- 
gest information-based businesses, 
collecting and analyzing open- 
source information. Open source 
has long been a high IC priority. 
Today open-source information has 
become a major challenge to the 
IC. The IC response to the chal- 
lenge is very much a dynamic 
work-in-progress . 

Open-source information is not 
what it used to be. Ten years ago, 
"open source" generally meant 
information from foreign newspa- 
pers and the electronic media, 
which was collected mostly by the 
Foreign Broadcast Information Ser- 
vice (FBIS). Open source was 
"frosting on the cake" of intelli- 
gence material dominated by 
signals, imagery, and human-source 
clandestine collection. 

Today, open source has expanded 
well beyond "frosting" and com- 
prises a large part of the cake itself. 
It has become indispensable to the 
production of authoritative analy- 
sis. It increasingly contains the best 
information to answer some of the 
most important questions posed to 
the Community. Media reports now 
are just a small piece of the open- 
source pie, which comprises a vast 
array of documents and reports 
publicly retrievable but often diffi- 
cult to find in today's high volume, 
high-speed information flow. Open 
sources provide vital information 
for the policymakers, whom the IC 
serves. Accessing, collecting, and 



analyzing open-source informa- 
tion, in short, is a multi-faceted 
challenge that can only be met with 
a multi-front response or strategy. 

This article examines three aspects 
of the open-source challenge/ 
response dynamic: its critical 
importance; how the IC is using 
technology to help the analyst cope 
with the information glut; and the 
need for interaction with the pri- 
vate sector. 



Critical Importance of Open 
Source 

The world for the IC analyst has 
changed dramatically since the end 
of the Cold War. A decade ago glo- 
bal coverage largely meant a 
comprehensive strategy to collect 
against and analyze the Soviet 
Union — the IC's single strategic 
threat. Today, global coverage 
entails the responsibility to assess 
diverse, complex, and dispersed 
threats around the world. In addi- 
tional to traditional intelligence 
concerns — such as the future of 
Russia and China, political turmoil 
in Indonesia, and civil conflicts in 
Africa — the new environment fea- 
tures many nontraditional missions. 
The IC now provides intelligence 
about peacekeeping operations, 
humanitarian assistance, sanctions 
monitoring, information warfare, 
and combating international orga- 
nized crime, as well as placing 
greater emphasis on such trans- 
national issues as counterterrorism, 
counternarcotics, and counterprolif- 
eration. Many of these missions are 
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operationally focused, requiring 
growing proportions of the ana- 
lytic and collection work force to 
function in an ad hoc crisis mode. 

Open-source information now 
dominates the universe of the intel- 
ligence analyst, a fact that is 
unlikely to change for the foresee- 
able future. The revolution in 
information technology and tele- 
communications has fundamentally 
transformed the globe that the IC 
covers, the services that it provides 
to consumers, and the workplace in 
which its people function. 

• Information abounds. A growing 
volume of open-source material is 
relevant to intelligence needs. 
Closed societies in the former 
Soviet Union and Eastern Europe 
have opened up, and reliable 
information now is widely avail- 
able. Fifteen years ago, 
information on the Balkans was 
scarce, foreign newspapers took 
weeks to arrive at an analyst's 
desk, and policymakers were will- 
ing to wait days or even weeks 
for a paper on the issues. 

• Today everything moves faster, 
and people are better informed. 
The revolution in information 
technology has vastly increased 
the volume and speed of the 
information flow across the globe 
and across computer screens. 
Technology makes analysts more 
efficient, but it also increases the 
demand from consumers. Intelli- 
gence requirements tend to be 
sharper and more time sensitive. 
Analysts often receive newspaper 
and media reports before the 
people in the countries where the 
reports are generated and 
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Automated analysis 
tools — data mining and 
retrieval techniques — 

provide significant 
opportunities to help 
solve the information 
overload problem. 

intelligence consumers will not 
tolerate waiting days for a 
response. 

• Governments have less and less 
capacity to control information 
flows. International organized 
crime groups, terrorists, narcotraf- 
fickers, and weapons proliferators 
are taking advantage of the new 
technologies, bypassing govern- 
ments or seeking to undermine 
them when governments try to 
block their illegal activities. 
Chances are these criminal net- 
works will be using laptop 
computers, establishing their own 
websites, and using sophisticated 
encryption. Tn the years ahead, 
these enhanced capabilities will 
raise the profile of transnational 
issues that are already putting 
heavy demands on intelligence 
collection and analysis. 



Dealing with the Information 
Glut 

The enhanced speed of communica- 
tion is a distinct advantage in 
today's world where intelligence 
analysts are as comfortable tn 
cyberspace as in the office space of 
top consumers. The Washington- 
based analyst can send a message 
and get a response from a post in a 
remote country 7 faster than it used 
to take to exchange notes by 



pneumatic tube with counterparts 
in the same building. 

The expanding volume of open- 
source information, however, pre- 
sents a greater challenge. During 
the Cold War, the job of the IC was 
to piece together bits of secret 
information. Each piece of raw 
intelligence was a carefully 
acquired golden nugget. Today, the 
IC is still mining for information but 
facing an avalanche from both 
open-source and classified collec- 
tion systems. 

Technology is a major part of the 
answer to the magnitude of the 
open-source challenge, but it is no 
substitute for the other essential 
component; skilled people. The IC 
must invest more in technology to 
provide the analytic tools needed to 
assess and exploit the vast amount 
of information available, and it 
must invest more in people, whose 
expertise is crucial for prioritizing, 
interpreting, and analyzing this 
information. The greater the vol- 
ume of information to assess, the 
stronger must be the expertise to 
evaluate it. 

The number of sources and the 
overall amount of data to which an 
analyst has access make the pro- 
cess of finding precise information 
or hidden clues extremely difficult. 
How can the analyst know where 
to start looking? What data might 
be relevant and what should be 
ignored? When intelligence ana- 
lysts query databases, they need to 
know how to ask the questions in a 
way that will get useful answers, 
and they need analytic tools to help 
them extract the right data. Auto- 
mated analysis tools — data mining 
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and retrieval techniques— provide 
significant opportunities to help 
solve the information overload 
problem. 

Cognitive analytic tools are under 
development in both the private 
sector and the government to facili- 
tate management of the information 
glut, enhancing the IC's ability to 
filter, search, and prioritize poten- 
tially overwhelming volumes of 
information. 

• Clustering lets analysts exploit the 
most useful data sets first, help- 
ing the IC perform its warning 
function. Clustering is particularly 
helpful when the volume of infor- 
mation, as with open source, 
makes It difficult to recognize 
meaningful patterns and 
relationships. 

• Link analysis helps to establish 
relationships between a known 
problem and unknown actors and 
detect patterns of activities that 
warrant particular attention. 

• Time-series analysis can enable 
analysts to track actions over time 
so that unusual patterns of activ- 
ity can be identified. 

• Visualization allows analysts to 
see complex data— including link 
and time-series analysis— laid out 
in new and varied formats that 
promote analytic insight. 

• Automated database population is 
designed to free analysts from the 
tedious and time-consuming func- 
tion of manually inputting 
information into databases, reduc- 
ing the potential for errors and 
inconsistencies. 
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FBIS is developing a 
single, open-source 
'portal'... to be accessible 
from desktops and 
expected to be fully 
operational by 2002. 

One of the strongest and most con- 
sistent needs of IC analysts is to 
search and exploit both classified 
and unclassified information from a 
single workstation. The Commu- 
nity is working on this and on ways 
to standardize information and tag 
it using metadata — or reference 
information—to make it easier to 
search, structure, and enter infor- 
mation into databases. 

FBIS is developing a single, open- 
source "portal" that will organize 
and cross-reference FBIS products, 
information that FBIS has collected 
via the Internet as well as other 
multimedia material. 

•The portal, to be accessible from 
desktops and expected to be fully 
operational by 2002, will provide 
analysts with a one-stop shop for 
all open-source intelligence, 
whether collected by FBIS or not. 

• Material on the portal will be 
indexed, archived, and accessible 
via a powerful, easy-to-use search 
engine. 

Enhancing IC Cooperation 

Collaborative tools offer a critical 
opportunity for enhanced coopera- 
tion among the IC's 13 agencies, 
DCI centers, the National intelli- 
gence Council, and literally 
hundreds of collection and analysis 



offices. The problem of sharing 
data among such a large number of 
organizations is immense, in partic- 
ular because different agencies 
have different security standards. 
Each organization has private intel- 
ligence holdings that are 
extraordinarily sensitive. The IC has 
to resolve the issue of multilevel 
security and need-to-know con- 
cerns by developing robust and 
flexible communities of interest 
using collaborative tools. 

New tools are needed to enhance 
cooperation in two areas: 

* Collaboration in the production 
process to increase speed and 
accuracy. 

• Expertise-based collaboration — to 
enable a team of analysts to work 
on a project for several weeks or 
months. 

Several collaborative tools cur- 
rently available or soon to be 
deployed include the capability to 
share both textual and graphic 
information in real time. These new 
tools will allow analysts to discuss 
contentious analytic issues; share 
information like maps, imagery, 
and database information; and 
coordinate draft assessments. This 
would all be done on line, from 
their own workspaces, resulting in 
substantial savings of time and 
effort over current practices. Future 
requirements emphasize broad 
deployment of collaborative tools, 
relying on mature commercial off- 
the-shelf platforms performing to 
standards that allow interoperabil- 
ity across the IC. 
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Another important aspect of 
enhanced collaboration is distrib- 
uted knowledge. The IC will never 
have a database that contains all 
information available to ail organi- 
zations, due to the individual 
missions of each organization. The 
ability to share major holdings of 
multiple agencies and to present an 
integrated view to the analyst's 
desktop, however, is critical and 
possible — but no easy task! 

Finally, the IC has some challenges 
that few private sector organiza- 
tions face. It deals, for example, 
extensively in foreign languages — 
lots of them. FBIS translates and 
disseminates information in many 
languages. Automated translation 
toois are getting better but still do 
not function adequately. The IC 
remains heavily dependent on 
trained linguists. 



Working With the Private Sector 

The information technology (IT) 
relationship between the US Gov- 
ernment and industry has 
undergone a dramatic transforma- 
tion in recent years. By itself, the IC 
simply cannot stay ahead of the 
technological curve and it knows it. 
Today, government no longer dom- 
inates research and development 
(R&D) and the information market- 
place — the private sector does. The 
IT industry's R&D is focused prima- 
rily on commercial applications; the 
IC's requirements increasingly will 
have to be satisfied by products 
developed for commercial use. The 
IC needs to develop close and 
enduring partnerships in the com- 
mercial world to benefit from both 
the private sector's continuing 
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The ability to share major 

holdings of multiple 
agencies and to present 
an integrated view to the 

analyst's desktop is 
critical and possible — but 
no easy task! 

pursuit of new technology and its 
best practices in dealing with the 
open-source challenge. 

In 50 years, the IC has gone from 
large, stationary mainframes with a 
handful of dumb workstations to 
portable multi-service devices that 
will communicate, compute, and 
run offices. This represents a dra- 
matic leveling of information costs 
and affects the way the IC does its 
work. In many ways, however, the 
Community still thinks and orga- 
nizes itself with immobile 
information systems. It is continu- 
ing to invest great amounts in 
stationary hardware systems, while 
many of its targets — terrorists, nar- 
cotraffickers, and organized crime 
syndicates — are becoming increas- 
ingly mobile in their operations. 
Perhaps private industry will come 
up with ways to liberate analysts 
from their cubicles, while at the 
same time ensuring the security of 
their work. 

The CIA has developed two organi- 
zations to build and sustain outside 
partnerships: the Office of 
Advanced Intelligence Tools (AIT) 
and In-Q-Tel. 

• In 1997, CIA's Directorate of Sci- 
ence and Technology and its 
Directorate of Intelligence collab- 
orated in the. formation of AIT. 
The office works inside CIA with 



analysts to determine their needs 
and outside CIA with vendors to 
identify state-of-the-art cognitive 
and collaborative tools. 

• CIA launched In-Q-Tel in 1999 as 
a nonprofit corporation designed 
to bring together the best of the 
academic, business, and private 
research worlds to exploit new 
and emerging information tech- 
nologies. 1 Its mission is twofold: 
first, to accept strategic problems 
and develop a "portfolio" of inno- 
vative IT solutions, ranging from 
exploration to demonstration; 
and, second, to fuel private 
research, development, and appli- 
cation of information technologies 
of strategic national interest for 
the benefit of all partners. 

In-Q-Tel is not designed to con- 
duct research itself; rather, it will 
orchestrate the work of numerous 
partner organizations working in 
teams. In-Q-TePs initial projects 
focus on four interrelated intelli- 
gence challenges: 

• Agency use of the Internet^ partic- 
ularly Internet search and privacy 
issues. 

• Information security, a crosscut- 
ting issue that permeates all 
organizational functions. In-Q-Tel 
will engage information security 
from the perspectives of harden- 
ing and intrusion detection; 
monitoring and profiling of 

1 The new corporation was first launched in 
February 1999 as In-Q-It, bur changed its 
name to In-Q-Tel in December 1999 to pre- 
vent confusion with the financial software 
giant Intuit. "In-Tel" is self-explanatory, while 
the "Q" stands for technical innovation- 
derived from the James Bond character who 
developed Bond's spy gear. 
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information use and misuse; and 
network and data protection. 

* Analytic data processing capabili- 
ties, including geospatial and 
multimedia data fusion/integra- 
tion, all source analysis, and 
computer data forensics. 

• Distributed information technol- 
ogy infrastructure, to facilitate 
data dispersal to multiple organi- 
zations/agencies anywhere in the 
world. 

The IC leadership recognizes that 
partnerships with outside technical 
and academic experts, as well as 
vendors, are essential to enabling 
us to stay on top of the informa- 
tion technology curve. Among 
analysts, the attitude and behavior 
toward the outside world is slowly 
changing, but the IC needs to pro- 
vide more incentives for analysts to 
get out from behind their desks to 
engage with substantive experts 
and other outside sources of use- 
ful — and increasingly critical — 
information that cannot be cap- 
tured by clandestine collectors or 
traditional open-source collectors 
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By itself, the Intelligence 

Community simply 
cannot stay ahead of the 
technological curve and 
it knows it. 

such as FBIS. This is an imperative, 
not an option. It has been said that, 
"Opportunities are like sunrises. If 
you wait too long, you miss them/' 
The iC cannot afford to miss 
today's opportunities because it is 
too inwardly focused. It does not 
intend to do so. 



Conclusion 

For most of its history, the Intelli- 
gence Community has operated as 
an industrial enterprise, with com- 
partmentation as a key operating 
metaphor. In the process, a set of 
impressive organizations has been 
created; however, they are now 
being overtaken by events. In the 
post-industrial world pervaded by 
information technology, networks 
defeat hierarchies, and agility 
becomes a prerequisite for organi- 
zational success. Even with the 



impressive gains of the past few 
years, dealing with the open-source 
challenge will necessarily be a 
work-in-progress for some time to 
come. Open source is not a tradi- 
tional collection challenge, and 
there is no single solution. Meeting 
the challenge requires a multi-front 
strategy, and it will take time for 
the IC to get this right. 

The IC recognizes that it can suc- 
ceed only if It exploits the changes 
taking place in the information rev- 
olution and in information 
industries. The Community always 
will have security concerns but can- 
not allow them to deter it from 
taking advantage of the opportuni- 
ties inherent in the emerging 
environment. 

The leadership has committed the 
IC to a corporate strategy that will 
leverage the best practices and 
resources of the whole govern- 
ment and the private sector to 
provide the President and US poli- 
cymakers the information 
advantage they need. 
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